MuMIC – Multimodal Embedding for Multi-Label Image Classification with Tempered Sigmoid

Multi-label image classification is a foundational topic in various domains. Multimodal learning approaches have recently achieved outstanding results representation and single-label classification. For instance, Contrastive Language-Image Pretraining (CLIP) demonstrates impressive image-text abilities robust to natural distribution shifts. This success inspires us leverage multimodal for multi-label tasks, benefit from contrastively learnt pretrained models. We propose the Image Classification (MuMIC) framework, which utilizes hardness-aware tempered sigmoid based Binary Cross Entropy loss function, thus enables optimization on objectives transfer CLIP. MuMIC capable of providing high performance, handling real-world noisy data, supporting zero-shot predictions, producing domain-specific embeddings. In this study, total 120 classes are defined, more than 140K positive annotations collected approximately 60K Booking.com images. The final model deployed Content Intelligence Platform, it outperforms other state-of-the-art models with 85.6% GAP@10 83.8% GAP all classes, as well 90.1% macro mAP score across 32 majority classes. summarize modelling choices extensively tested through ablation studies. To best our knowledge, we first adapt pretraining problems, innovation can be transferred